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Preface 


The  Institute  for  Defense  Analyses  (IDA)  prepared  this  document  for  the  Office  of  the 
Director,  Program  Analysis  and  Evaluation  under  a  task  titled  "Cost  Research 
Symposium."  It  contains  an  annotated  version  of  a  presentation  the  author  delivered  at  the 
third  annual  Life  Cycle  Costing  in  Defence  Conference,  held  on  17-18  June  2002  in  London, 
England.  Tire  conference  was  sponsored  by  the  LSC  Group,  TED  Services  &  Pennant 
Software  Services,  and  was  supported  by  the  International  Society  of  Logistics  (known  as 
SOLE).  Attendees  included  representatives  from  the  UK  Defence  Procurement  Agency,  the 
UK  Ministry  of  Defence,  the  UK  Defence  Export  Services  Organisation,  the  Norwegian 
Defence  Logistics  Organisation,  General  Dynamics,  BAE  Systems,  Eurocopter,  Thales,  Racal 
Instruments,  and  others. 

This  document  was  not  reviewed  within  IDA  for  its  technical  content.  The  views  and 
opinions  it  contains  are  attributable  solely  to  the  author. 


Cost  in  Cost  Effectiveness 

A  presentation  by 

Dr.  Stephen  J.  Balut 

Institute  for  Defense  Analyses 

at  the 

Life  Cycle  Costing  in  Defence  Conference 

17-18  June  2002 


I'm  Steve  Balut,  Director  of  the  Cost  Analysis  and  Research  Division  at  the  Institute 
for  Defense  Analyses  in  Alexandria,  Virginia.  The  Institute  for  Defense  Analyses,  or  IDA,  as 
we  call  it,  is  a  non-profit  corporation  that  provides  advice  to  the  Department  of  Defense  on 
matters  related  to  science  and  technology. 

IDA  assists  the  Department  of  Defense  by  conducting  cost-effectiveness  studies  of 
alternative  ways  of  performing  defense  missions.  The  subjects  of  several  recent  studies 
include  the  F-22  tactical  aircraft,  the  Joint  Strike  Fighter,  the  C-17  cargo  aircraft,  and  airborne 
electronic  attack  systems.  Cost-effectiveness  analysis  is  central  to  our  business.  When 
conducting  cost-effectiveness  analyses,  the  Cost  Analysis  and  Research  Division  of  IDA 
forecasts  the  life  cycle  costs  of  current  and  prospective  weapon  systems. 
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Introduction 

•  Cost-Effectiveness  Analysis,  an  analytical  technique 
routinely  applied  to  aide  in  selecting  the  best  way  to 
perform  defense  missions  under  constrained  budgets 

•  Role  of  Cost  in  Cost-Effectiveness  Analysis,  creates 
incentives  for  contractors  and  military  departments  that  can 
bring  unexpected,  undesirable  results 

•  Policing  Costs:  counters  adverse  incentives  and 
strengthens  integrity  of  the  process 


Everyone  here  is  familiar  with  the  fundamental  idea  of  cost-effectiveness  analysis.  This 
technique  provides  visibility  into  both  the  operational  and  economic  implications  of 
alternative  ways  of  performing  a  defense  mission.  It  is  important  to  note  that  such  analyses 
say  nothing  about  the  social,  distributional,  or  political  aspects  of  the  choice  at  hand.  These 
are  considerations  that  decision  makers  must  fold  into  their  thinking.  But  knowing  what 
each  alternative  can  do  and  what  it  costs  forms  a  sound  basis  for  proceeding  with  a  choice 
among  competing  alternatives. 

The  cost  part  of  cost-effectiveness  provides  visibility  and  understanding  of  the  costs 
that  will  be  borne  if  an  alternative  is  selected.  All  other  things  being  equal,  the  lower  the  cost 
of  an  alternative,  the  more  efficient  it  appears  and  the  more  likely  it  is  to  be  selected  and 
funded.  The  rub  here  is  that,  in  many  cases,  proponents  of  an  alternative— that  is,  contractors 
and  military  departments— help  prepare  the  cost  estimates.  This  process  can  create 
incentives  that  lead  to  undesirable  results. 

Decision  systems  that  fail  to  deal  directly  with  this  natural  but  destructive  incentive  are 
doomed  to  failure.  The  effects  are  loss  of  capability  and  disruptive  programming  and 
budgeting.  This  problem  can  and  has  been  countered  by  policing  cost  estimates.  I  will  have  a 
lot  to  say  about  how  to  maintain  the  integrity  of  the  decision  process  by  policing  cost 
estimates  at  the  appropriate  place  and  time. 
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Cost-Effectiveness  Methodology 

•  Identify  mission  and  alternative  ways  of  accomplishing  it 

•  Estimate  effectiveness  of  alternatives 

•  Estimate  costs  of  alternatives 

•  Combine  costs  and  effectiveness  to  reveal  preferred 
alternatives 

•  List  assumptions  and  limitations  of  analysis 


Just  to  make  sure  we're  all  talking  about  the  same  thing,  I  will  run  down  the  steps  involved 
in  conducting  a  cost-effectiveness  study. 

The  problem  starts  with  identification  of  a  defense  mission,  such  as  moving  war  fighters 
and  their  equipment  from  their  bases  in  one  part  of  tire  world  to  a  point  of  battle  in  another. 
Alternatives  are  identified  for  accomplishing  the  objective.  In  this  example,  alternatives  could 
include  different  mixes  of  aircraft,  ships,  and  land  vehicle  systems. 

Tire  effectiveness  of  each  alternative  is  then  estimated.  This  is  accomplished  by  calculating 
measures  of  effectiveness  that  are  representative  of  the  task.  Simulation  is  a  common  way  to  do 
this  step.  In  the  example  of  moving  forces  to  a  point  of  battle,  a  measure  that  has  been  used  is  ton- 
miles  delivered  over  some  fixed  period  of  time. 

Tire  next  step  is  to  develop  costs  of  each  alternative.  This  involves  full  life  cycle  costs  of 
systems  involved  along  with  other  relevant  costs. 

These  costs  are  then  combined  with  measures  of  effectiveness  in  a  way  that  reveals  the 
preferred  alternatives.  In  the  example,  tire  preferred  alternative  might  be  the  one  that  delivers  a 
given  amount  of  ton-miles  per  day  over  some  period  of  time  for  tire  least  cost. 

Finally,  the  analyst  owes  the  decision  maker  an  explanation  of  tire  assumptions,  data 
sources,  uncertainties,  and  limitations  of  the  analysis. 
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Estimating  Costs 


Cost  Effectiveness 
Preferred  Alternative 

Effectiveness 

Force  Integration 
Tactics 

Life  Cycle  Costs 

Manning 

Basing 

Maintenance 

Operations 

Performance 
Acquisition  Costs 

Development 

Production 


This  slide  depicts  the  sources  and  flows  of  data  and  other  information  that  support  a  cost- 
effectiveness  analysis.  Industry  is  one  of  the  basic  sources  of  information  on  systems.  This 
information  includes  descriptions  of  physical  and  technical  features  of  the  systems,  the 
performance  of  the  systems,  and  forecasts  of  the  costs  of  developing  and  producing  the  systems. 
Industry  works  closely  with  the  prospective  buying  military  departments,  providing  cost 
estimates  for  proposed  contract  activities. 

The  military  departments  work  with  industry  to  add  operating  and  support  cost  estimates 
based  on  the  concept  of  operations,  manning,  basing,  maintenance  policy,  and  other  factors.  This 
extends  the  industry  acquisition  cost  estimate  to  a  military  department  life  cycle  cost  estimate.  Tire 
military  departments  also  estimate  the  effectiveness  of  forces  that  include  the  systems  of  interest. 

Tire  Office  of  tire  Secretary  of  Defense  (OSD)  receives  these  inputs  from  industry  and  the 
military  departments,  combines  the  cost  and  effectiveness  estimates,  and  identifies  the  preferred 
alternatives. 

United  States  statute  requires  an  independent  cost  estimate  be  performed  as  a  weapon 
system  approaches  a  milestone  decision,  such  as  entrance  into  development  or  production.  That 
is,  an  estimate  must  be  prepared  by  an  organization  that  is  separate  from  the  buying  military 
department.  These  estimates  are  now  done  by  tire  OSD  Cost  Analysis  Improvement  Group,  a  part 
of  the  OSD  Staff.  However,  prior  to  about  a  decade  ago,  the  so-called  "independent  estimates" 
were  performed  by  the  buying  military  departments. 
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Role  of  Cost 

•  Fix  effectiveness  level:  least  cost  alternative  is  preferred 

•  Cap  costs:  alternative  with  greatest  effectiveness  is 
preferred 

•  Allow  both  cost  and  effectiveness  to  float:  indifference 
among  alternatives  along  “Efficient  Frontier” 


C 

Lower  cost  increases  chance  of  being  preferred  alternative 


The  role  of  cost  in  cost-effectiveness  analyses  varies  depending  upon  the  approach 
applied.  If  effectiveness  is  fixed  at  some  level,  then  the  least  cost  alternative  wins.  Another 
approach  is  to  cap  costs  and  calculate  which  alternative  achieves  the  highest  level  of 
effectiveness  under  this  budget  constraint.  A  third  way  to  conduct  the  analysis  is  to  allow 
both  costs  and  effectiveness  to  float,  calculate  both  for  all  alternatives,  then  array  these  data 
as  shown  in  the  graph  on  the  slide. 

The  graph  shows  which  alternatives  are  dominated  by  other  alternatives.  For 
example,  alternative  4  is  dominated  by  both  alternatives  2  and  3  because  the  effectiveness 
levels  of  both  2  and  3  are  higher  than  4,  and  4  costs  more  than  either  2  or  3.  It  also  reveals 
those  alternatives  that  are  not  dominated— in  this  example,  alternatives  1,  3,  and  6.  These 
non-dominated  alternatives  array  themselves  in  an  "efficient  frontier"  pattern.  That  is,  they 
tend  to  create  a  frontier  along  which  the  decision  maker  should  be  indifferent  to  which  is 
selected  from  a  cost-effectiveness  perspective.  At  this  point,  the  decision  maker  can  make 
his  choice  based  on  affordability,  minimum  desired  effectiveness  level,  or  other  factors.  In 
practice,  once  the  frontier  is  identified,  additional  alternatives  can  be  constructed  near  the 
frontier  at  points  of  particular  interest. 

The  bottom  line  is  that  no  matter  the  approach,  alternatives  with  lower  cost  are  more 
likely  to  be  the  preferred  alternative. 
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Incentives  to  “Buy-In” 

•  Programs  once  started  rarely  get  cancelled 

-  Incentive  to  get  “foot  in  the  door” 

-  Deal  with  reality  later 

•  Win  can  establish  monopoly  position 

•  Early  “learning”  enhances  competitive  advantage  for 
future  contracts 

•  Future  contract  modifications  offer  opportunities  to 
recover  from  buy-in 


Following  this  process,  both  industry  and  the  buying  military  department  have  strong 
incentives  to  provide  cost  estimates  that  are  optimistic.  Our  experience  shows  these  early 
optimistic  estimates  are  rarely,  if  ever,  achieved.  Skeptics  of  this  process  say  proponents 
lowball  early  estimates  to  get  their  foot  in  the  door,  or  nose  under  the  tent,  intending  to  deal 
with  expected  cost  growth  later,  after  the  Department  of  Defense  is  committed  to  the 
system. 

Winning  a  competition  puts  the  contractor  in  an  advantageous  position.  It  could 
establish  a  monopoly  position  for  the  contractor,  enhancing  his  negotiating  position  for  later 
lots  in  the  acquisition.  In  addition,  we  all  know  that  the  early  units  are  the  most  expensive 
due  to  the  "cost  progress"  or  "learning"  phenomenon.  Once  down  the  steep  part  of  the 
learning  curve,  the  winning  contractor  has  a  distinct  advantage  over  other  contractors  who 
must  start  back  up  the  learning  curve  at  high  cost  unit  one. 

The  real  ace  in  the  hole  for  the  winning  contractor  is  full  knowledge  that  the  system 
will  evolve  with  time  as  technology  advances.  The  government  will  come  back  to  the 
contractor  requesting  modifications  to  the  original  contract  to  add  capabilities  facilitated  by 
such  advances.  Once  the  contract  is  reopened,  cost  estimates  are  revised,  providing 
repeated  opportunities  to  recover  from  early  lowball  estimates. 
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Damage  of  Cost  Overruns 

•  Budgets  squeezed 

-  Program  stretch-outs,  delays,  cancellations 

-  Financial  plans  disrupted 

•  Result  in  reduction  in  capability 

-  Fewer  systems 

-  Systems  arrive  later  than  needed 

-  Other  capability  may  be  delayed  or  cancelled  to  pay  for 
overrun 

Warfighters  must  adapt  to  reduced  capabilities 


Given  the  factors  just  discussed,  cost  growth  on  major  weapon  system  acquisitions 
seems  inevitable.  In  spite  of  repeated  episodes  of  massive  cost  growth,  our  budgeting 
process  continues  to  join  in  the  early  optimism  of  the  buying  military  departments  who 
prepare  low  budget  estimates.  You've  heard  the  phrase  "keep  their  feet  to  the  fire."  In 
practice,  that  means  budget  to  the  low  estimate  and  see  if  they  can  pull  it  off.  They  usually 
cannot. 

Cost  growth  causes  serious  budget  problems.  Program  stretch-outs  and  delays  result 
in  fewer  weapon  systems  entering  the  force.  Cancellations  are  rare,  but  they  do  occur. 
Examples  include  the  A-12  and  the  Crusader.  Systems  that  do  enter  the  force  do  so  later 
than  planned,  sometimes  with  reduced  capabilities.  Further,  paying  for  cost  overruns  takes 
dollars  away  from  other  planned  acquisitions.  That  means  even  fewer  weapons  and  later 
capabilities,  even  in  areas  not  experiencing  cost  growth. 

In  the  end,  cost  growth  translates  to  a  reduced  warfighting  capability. 
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Policing  Costs 


Cost  Effectiveness 
Preferred  Alternative 

Effectiveness 

Force  Integration 
Tactics 

Life  Cycle  Costs 

Manning 

Basing 

Maintenance 

Operations 

Performance 
Acquisition  Costs 

Development 

Production 


Independent,  objective  filter 

Alternatives 

Navy  )  (AirForce 


Systems 


Industry 


What  can  be  done  to  police  cost  estimates  flowing  from  industry  and  the  military 
departments?  I  showed  this  diagram  previously,  but  I've  inserted  a  new  box  between  the 
military  departments  and  OSD.  This  box  represents  a  point  in  the  process  where  cost 
estimates  and  related  information  can  be  reviewed  for  realism  and  completeness.  Let's 
compare  this  placement  to  where  the  filter  has  been  located  in  the  past. 

In  the  early  1970s,  the  military  departments  performed  this  filter  function  in  the  U.S. 
Department  of  Defense.  In  terms  of  the  diagram,  this  would  have  the  new  box  down  at  the 
location  of  the  military  departments.  Placing  the  filter  function  there  does  nothing  to  solve 
or  counter  the  perverse  incentives  problem.  Since  the  early  1990s,  the  filter  function  has 
been  performed  within  the  Office  of  the  Secretary  of  Defense,  many  times  with  the 
assistance  of  organizations  like  IDA.  This  placement,  shown  in  the  diagram,  can  work 
depending  upon  the  nature  of  the  filter. 
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Filter 

•  Independent,  objective 

•  No  conflict  of  interest — arm’s  length  relationship  with 
advocates 

•  Highly  capable 

-  Latest  and  “best”  methods 

-  Access  to  industry  and  government  cost  experiences 

-  Knowledge  of  trends  in  technology  and  cost 

•  Experience 

-  Participation  in  programming,  budgeting,  and 
acquisition  processes 

-  Awareness  of  common  pitfalls 


The  group  that  performs  the  policing  function  must  be  independent  and  objective. 
That  is,  it  must  (1)  have  no  vested  interest  in  the  outcome  of  the  pending  decision  and  (2)  be 
free  of  any  possible  bias  towards  any  of  the  alternatives.  Further,  the  group  should  be  free 
from  conflicts  of  interest,  particularly  with  industry,  and  maintain  an  arm's  length 
relationship  with  advocates  in  the  military  departments. 

These  conditions  are  necessary,  but  not  sufficient.  The  filtering  organization  must  also 
be  highly  capable— preferably  the  best  in  the  business  of  defense  cost  estimating— and  use 
the  latest  and  best  methods.  While  maintaining  independence  and  objectivity,  the  group 
must  have  ready  access  to  both  industry  and  government  cost  experiences  and  be 
knowledgeable  of  trends  in  both  technology  and  costs. 

Further,  the  group  must  be  experienced  in  programming  and  budgeting  and  have  a 
full  understanding  of  the  acquisition  process.  They  need  to  know  where  to  look  in  complex 
budgets  to  identify  the  funding  streams.  Prior  experience  with  funding  sleights  of  hand  is 
also  helpful. 
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Realistic,  Comparable  Costs 

•  Realistic 

-  Consistent  with  historical  costs  for  similar  systems  and 
operations 

-  Alignment  with  trends  in  both  technology  and  cost 

•  Comparable 

-  Estimate  probability  distribution  of  life  cycle  cost  for 
each  alternative 

-  Use  estimates  of  life  cycle  cost  at  same  risk  level  for  all 
alternatives 


Policing  cost  estimates  means  ensuring  that  they  are  realistic,  complete,  and 
comparable  across  all  alternatives.  One  test  for  realism  is  to  look  for  consistency  with  cost 
experiences  on  analogous  systems  and  operations  while  taking  differences  into  account. 
This  test  can  be  strengthened  by  placing  estimates  along  trend  lines  that  display  measures 
of  technological  progress  along  with  the  costs  of  achieving  this  progress.  An  example  of  this 
technique  would  be  to  plot  the  increasing  trend  in  cost  associated  with  increasing  thrust-to- 
weight  ratios  achieved  by  aircraft  engines.  This  trend  line  could  be  used  to  forecast  the 
increase  in  cost  that  might  be  expected  from  yet  another  proposed  advance  in  engine  thrust- 
to-weight. 

Policing  includes  ensuring  that  costs  of  alternatives  are  comparable.  They  must 
include  and  exclude  things  in  a  consistent  way.  Uncertainties  should  be  dealt  with  directly 
by  estimating  probability  distributions  of  costs  and  presenting  estimates  at  the  same  risk 
levels. 
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Effects  of  Policing  Costs 

•  More  credible  cost  estimates 

•  Fewer  and  smaller  cost  overruns 

•  Less  disruptive  budget  process 

•  Greater  confidence  in  system 


Less  loss  of  warfighting  capabilities 


Effective  policing  of  costs  at  the  right  point  in  the  process  pays  big  dividends.  It  will 
not  remove  incentives  to  lowball  estimates.  Industry  and  advocates  in  the  military 
departments  will  continue  to  suffer  from  this  affliction.  But,  policing  will  make  lowball  cost 
estimates  less  likely.  Advocates  will  know  someone  is  watching.  Proposals  with  unrealistic 
cost  estimates  will  not  pass  muster.  More  credible  cost  estimates  will  be  demanded  before 
allowing  acquisition  programs  to  pass  through  major  milestones. 

The  initial  impact  of  policing  will  likely  be  higher  initial  estimates  for  systems  and 
operations,  but  the  downstream  effect  should  be  fewer  and  smaller  cost  overruns,  less 
frequent  program  stretches,  smaller  reductions  in  program  quantities,  and  fewer  program 
cancellations  due  to  unexpected  hemorrhaging  of  costs.  This,  in  turn,  results  in  less 
disruption  in  budgeting,  greater  confidence  in  the  system,  and  better  planning  overall. 

The  greatest  benefit  to  a  strengthened  process  is  getting  sufficient  warfighting 
capability  where  it  is  needed  when  it  is  needed,  rather  than  not  enough  and  too  late. 
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Summary 

•  Cost-effectiveness  intended  to  maximize  warfighting 
capabilities  under  constrained  defense  budgets 

•  Cost-effectiveness  methodology  prefers  least  cost 
alternatives 

•  Incentives  to  “buy-in”  can  lead  to  unintended  effects: 
reductions  in  warfighting  capabilities 

•  Adverse  incentives  can  be  countered:  police  process  by 
implementing  independent,  objective  filter  to  ensure  costs 
are  realistic  and  consistently  applied  across  all  alternatives 


In  summary,  cost-effectiveness  analysis  has  been  applied  to  aid  in  the  selection  of 
efficient  ways  to  perform  defense  missions  under  constrained  budgets.  The  logic  of  the 
analysis  technique  is  sound.  However,  the  role  of  cost  in  the  decision  process  creates 
incentives  that  can  lead  to  undesirable  results.  Past  failures  of  the  process  can  be  traced  to 
defective,  optimistic  costing  of  systems  and  alternatives.  Tire  incentive  to  buy  in  is  strong. 
Near-term  rewards  for  winning  are  substantial.  However,  the  long-term  effects  degrade 
warfighting  capabilities  and  disrupt  defense  management. 

We  can  do  better.  Adverse  incentives  created  by  the  role  of  cost  in  cost-effectiveness 
analysis  cannot  be  avoided,  but  they  can  be  countered.  We  can  police  the  costs  flowing 
through  the  system  acquisition  process  by  inserting  an  independent,  objective  filter.  This 
will  result  in  a  more  orderly,  efficient,  and  credible  system.  The  damaging  effects  of  cost 
overruns  can  be  reduced,  leading  to  improved  warfighting  capabilities. 
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